Digital data communication is fast becoming a ubiquitous aspect of modern life. The performance of devices relying on such communications continually improves at an ever-increasing pace. As a result, there is an ever-increasing demand for communications services and an attendant increase in the bandwidth that the data communications facilities must provide.
Modern data communications utilize a variety of different forms of packet communications. Within a packet network, many of the links are point-to-point links between peer nodes. When a need for increased bandwidth on such a link arises, one option is to replace the existing link with a higher bandwidth link. However, this approach often is prohibitively expensive, particularly as an incremental upgrade. Accordingly, the solution to providing increased bandwidth often entails deploying one or more new links of similar or somewhat higher capacity in parallel to the existing link between the nodes forming the two end points. Such an upgrade forms a multilink bundle, with an aggregate bandwidth that is essentially the sum of the bandwidths or throughput rates of the parallel links in the bundle.
The multilink bundle does offer increased capacity, however, its use presents certain problems. To effectively utilize the links in the bundle as an aggregate requires that the bundle effectively appear as a single interface to each end-point node. At one logical level, all traffic intended for transport through this interface must appear as a single, unified data stream. However, within the bundle, it is necessary to segregate the traffic and distribute segments of traffic to the different links.
To maximize utilization of the parallel links within a multilink bundle, the packet communication load on the two or more parallel links must be “balanced,” that is to say the volume of packet data traffic through the bundle must be optimally distributed between the various parallel links. A number of known algorithms are used to select which link to use to transport the various packets between the end-points and thereby balance the load on the links.
One type of approach manages the packet routing over the links based on link utilization. For example, when traffic on a first link reaches a certain level, the sending node shifts further packet traffic over to the next parallel link. This approach tends to load one link first, then another, and so on. Another general class of load-balancing techniques evenly distributes packets across all the links regardless of the current level of utilization of any particular link. This later approach reduces packet latency. Within this class, there are a number of specific approaches to load balancing.
One approach comprises Internet Protocol (IP) flow-based balancing. A flow is a unique conversation between two terminations on the data network. Flow-based balancing assigns each flow to one link. Once a flow is designated to use a particular link, it always goes over that link. A new flow may go over another link.
With most implementations of the flow-based balancing technique, a router or the like looks at the header of each packet and hashes all or part of the header. The IP flow-based balancing algorithms may hash different amounts of information from the header. Typically, the algorithms form a hash from the source address, the source port, the destination address and the destination port, i.e. the data from the header that uniquely defines a particular flow.
The router sends all packets with the same hash code over the same link in the bundle. The packets with the same header information produce the same hash code, and the router always assigns the packets with that hash to use the same link. Packets for communications having different headers, for example for a new session between a different source and destination, receive a different hash and may go over a different link in the point-to-point bundle. However, all packets in any one flow traverse only one link, in exactly the order received by the router. There is no need to add a sequence header or rearrange packets to compensate for different transit times over different links.
Flow-based balancing distributes flows over multiple links. However, no one flow can use all the bandwidth of all the links. If a single flow were to use more than one link, the packets may get reordered because of different packet lengths or different link latencies. There is nothing in the IP link layer to correct the order. TCP will recover packet order. However, reordering increases delays and reduces throughput. For example, it may drop a frame and request a retransmission. Hence, each flow must use a single link within the bundle. Consequently, one flow at most can access the maximum bandwidth of the largest of the channels or links within the bundle.
Many point-to-point links today use PPP, the Point-to-Point Protocol. The Multilink Protocol (RFC 1717) is a standardized extension of the PPP, which enables combination of PPP links or channels into a ‘Multilink bundle’ for higher rate point-to-point communications. The Multilink Protocol (MLP) uses packet fragmentation and a round-robin link assignment technique to distribute portions of data traffic over the various links within a bundle.
A typical load-balancing situation using the Multilink Protocol involves a point-to-point bundle of links between two peers, for example, between two routers. A common implementation may use multiple ISDN links between the nodes, for example basic rate or primary rate (T1) ISDN connections.
A router at one end of the bundle separates a long IP packet into two or more fragments and adds an MLP sequence header to each fragment. The load-balancing algorithm distributes the fragments taken from each packet over an appropriate number of the links, to allow parallel transmission through the bundle. In this manner, packets from a particular flow actually utilize any or all of the links in the bundle.
The communications over parallel links may encounter different delays in transmission. Fragments of one packet or succeeding packets may arrive at the receiver in a different order than when sent out over the multilink bundle. The PPP Multilink Protocol therefore uses packet sequencing to order fragments and packets. Specifically, a sequence number included in the header of each fragment of a packet allows the receiver to properly reorder the fragments and packets as they arrive over different links in the bundle.
The use of the sequence numbers, the differences in transit time through the links and the subsequent reordering all add delay and processing time. A goal is to make the bundle appear as a single point-to-point interface. In this regard it would be preferable to avoid reordering if possible.
The Multilink Protocol also provides for transport of two classes of packets, packets with sequence numbers and packets without sequence numbers. It is possible to expedite communication of a packet through the bundle by omitting the sequence number.
Using this class distinction under PPP, it becomes possible to interleave delay-sensitive traffic with packet fragments of other traffic. If the sending router receives a delay-sensitive packet, such as a voice over IP (VOIP) packet, the router can insert that packet (or fragments thereof) between the fragments of larger packets, that is to say by interleaving the delay sensitive packets between the fragments of the other packets. The delay-sensitive packet or its fragments do not receive sequence numbers. The interleaving approach works on a single point-point-point link, e.g. a single T1. However, because there is no sequence number, there is no way to handle the delay-bounded traffic over different links in a multilink bundle. The existing systems therefore have not been able to combine the fragmentation and attendant load balancing with transport of delay sensitive packets.
Per-flow load balancing techniques do not impose any additional delays due to differences in transit times through the multiple links or reorder processing caused by transport of packets of one flow over two or more links as in the Multilink fragmentation approach. However, existing per-flow balancing systems treat all traffic the same and impose the single link per flow limitation on all flows through the bundle.
Extensions to the Multilink Protocol RFC are in debate to add multiple sequence spaces to the multilink header sequence number. With these extensions, the bounded delay packets could be distributed over any link in the bundle because the receiver will guarantee that packets are received in the order sent. This proposed solution has a problem for bounded delay packets which may have fragments sent across more than one link in the bundle where the links have a large differential delay. Multilink bundles over high-speed links must be designed with 50-250 ms of differential delay because the links may traverse different geographies. Any bounded delay packets with fragments sent over these links would incur additional latency due to the differential delay.
A need therefore exists for systems and methodologies to allow transport of two or more classes of traffic through a multilink bundle, with appropriate treatments for optimal load balancing. Delay sensitive traffic should be distributed evenly over all links in the bundle, however, the system and methodologies must minimize delays and differences in delays imposed on communication of delay sensitive or delay-bounded traffic. Traffic that is not delay sensitive should be distributed evenly and in such a manner as to enable any particular communication or flow to utilize any or all of the available links, however, the combined load balancing technique should enable interleaving of the delay sensitive traffic as part of the efforts to minimize delays for such traffic.